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Abstract: Let R be an element of GL{n^M) having integer entries and let 
p : T" — denote the induced map on the torus T" = M"/Z". It is well 
known that p is ergodic with respect to the Haar measure on if and only 
if none of the eigenvalues of i? is a root of unity. Dani has shown that there 
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exists a subset S of such that for any x G S and any semisimple surjective 
endomorphism p of T" such that the corresponding hnear endomorphism 
has no eigenvalue on the unit circle, the closure of the orbit {p^{x)\k > 0} 
contains no periodic points and that the set S is 'large' in the sense that for 
any nonempty open set U of T" the set f/ fl S* has Hausdorff dimension n. In 
this paper, we shall prove an analogous result for certain endomorphisms of 
nilmanifolds and infranil manifolds. 

1 Introduction 

Let be a simply connected nilpotent Lie group. Assume that has a 
discrete subgroup A so that M = N/A is compact, (cf. Theorem 2.12 and 
Remark 2.14, g.) Then A n A^\ where A^^ = [A^, A^], is a uniform lattice 
in A^^ Let T = N/A = N/{N^ ■ A) where N = N/N^ is the abelianization 
of A^ and A is the image of A under the natural projection A^ — >N. One 
has a smooth bundle projection p : M — >T with fibre N^/A fl A^^ which is 
again a nilmanifold. We put the Riemannian metric on M induced by a right 
invariant metric on A^. 

Any automorphism Rof N such that -R(A) C A induces an automorphism 
R of N such that -R(A) (Z A. R and R induce self-maps p : M — >M and p : 
T — >T. The map p : M — >M is a bundle map covering the map p : T — >T. 
We call p an endomorphism of M. A theorem of W. Parry 0] says that p is 
ergodic if and only if p is, that is, if and only if none of the eigenvalues of 
R : A^ — i>A^ is a root of unity. 

Definition 1. Let C be a collection of self-maps of a topological space X. 
We say that a nonempty subset S G X is exceptional relative to C if the 
following condition holds: for any f & C and any x G S*, the closure of the 
/-orbit {f''{x)\k > 0} does not contain any /-periodic point. 
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When a class C of self maps of a smooth Riemannian manifold M contains 
an ergodic map, one expects any exceptional set to be 'small'. Indeed such a 
set is necessarily of measure zero and is of first category, i.e, its complement 
contains a residual set. However, relative to the class of all semisimple surjec- 
tive endomorphisms of the standard ra-torus T" such that the corresponding 
linear endomorphisms of have no eigenvalue on the unit circle, Dani has 
proved the existence of an exceptional set C T" which is 'large' in the sense 
that for any nonempty open subset [/ of T", the set S r\U has Hausdorff 
dimension n. In fact, Dani shows that S is an a- winning set of the Schmidt 
game (cf. §2). 

The purpose of this note is to extend Dani's result to a certain class 
of endomorphisms of nilmanifolds and of infranil manifolds. Examples of 
nilmanifolds are U /T where U is the group of unipotent upper triangular n- 
by-n matrices over M (resp. C) and F is the subgroup of those having entries 
in Z, (resp. in Z[z]). Nilmanifolds occur naturally in Riemannian geometry. 
For example, they arise in the study of the geometry near a cusp in a finite 
volume (non-compact) Riemannian manifold whose sectional curvatures are 
bounded above by a negative constant. See for a detailed study of certain 
nilmanifolds arising this way. 

Let Cf denote the collection of all surjective endomorphisms of an n-torus 
T such that the corresponding linear automorphisms of M" are semisimple 
and have no eigenvalue on the unit circle. We prove 

Theorem 2. Let M = N/A be an n-dimensional nilmanifold. Let C he the 
set of all surjective endomorphisms p of M such that p G Cj. Then there 
exists a positive number /i < 1/2 and an exceptional set S d M relative to 
C which is a-winning for any a G (0, fi]. In particular, S (lU has Hausdorff 
dimension n for any nonempty open set U C M . 

Let be a simply connected nilpotent Lie group. Let Aff{N) denote the 
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group of all affine transformations of which acts on the right of A^, that 
is, Aff{N) is the group generated by the right translations by elements of 
and the automorphisms of A^. Let F be a subgroup of Aff{N) such that 
action of F on A^ is free and properly discontinuous with the quotient N/T 
being compact. The quotient M := N/T is called an infranil manifold. It is 
known that the subgroup A C F consisting of all right translations of A^ is 
a finite index normal subgroup. We shall identify an element of A with the 
image of the identity element e E N under it. Then it is a result of Auslander 
that A is a uniform lattice in A^. 

Denote by M the nilmanifold N / K. One has a finite normal covering 
TT : M — with deck transformation group G := F/A. Starting with the 
metric on M induced by the right invariant metric on A^, the averaging 
process leads to a G-invariant metric on M . We put this G-invariant metric 
on M and the induced metric on M so that vr is a local isometry. 

Let R : A^ — >N be an automorphism which preserves the F action, that 
is, for any x (z N and 7 G F, there exists a 7' G F such that R{x'y) = 
R{x)'y'. Then -R(A) C A. R induces a self-map p : M — >M which we call an 
endomorphism of M. Note that p is covered by the endomorphism p of M 
defined by R. We are now ready state 

Theorem 3. Let C be the collection of all endomorphisms p of M induced by 
automorphisms R : A^ — >N such that the corresponding toral endomorphism 
J) : T — >T is in Cj. Then there exist a positive number u < 1/2 and an 
exceptional set S relative to C which is a-winning for any a G (0, u] . In 
particular, for any nonempty open set U C M , the set S nU has Hausdorff 
dimension n. 

Remark: Theorem ^ includes an important class of manifolds, namely, com- 
pact connected flat Riemannian manifolds; these are the infranil manifolds 
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when the simply connected nilpotent Lie group is abehan, i.e., = M". 
Then the compact flat manifolds are precisely those finitely covered by an 
ra-torus T". 

Our proofs use Dani's result and some basic observations about the na- 
ture of pre-image (resp. image) of a-winning sets under a smooth bundle 
projection (resp. a covering projection). 

Acknowledgements: We thank Professor S.G.Dani for his valuable com- 
ments on an earlier version of this paper. The authors gratefully acknowledge 
financial support from Department of Science & Technology, Government of 
India. 



2 The Schmidt game 



In this section we give a brief description of the Schmidt game [0 and recall 



here for the benefit of the reader some basic facts about winning sets. 

Let X be a complete metric space. Let < a, (3 < 1 and let S" be a 
subset of X. The Schmidt game on X is played by two players A and B 
as follows: B first chooses any closed ball Bq in X of radius r where r G M 
is positive. Then A chooses a closed ball Ai C Bq of radius ar. Then B 
chooses a closed ball Bi C Ai of radius ri = Par. Inductively, after B has 
chosen closed a ball B^ of radius = (a/5)'^r, k > 1, A chooses a closed 
ball Ak+i C Bk of radius ar^ and B chooses a closed ball Bk+i of radius 
Tfe+i = Park- Since limk^oork = 0, and since X is a complete metric space, 
^k>iAk is a singleton set {x}. We say that S is an (a, /?) -winning set (for 
A) if for any choices of B, A can always make her choices so that x & S. 
We say that S is a-winning if it is (a, /?)- winning for any f3 G (0, 1). Of 
course, the whole space X is an a- winning set for any a G (0, 1). Schmidt 
has shown that if /3 < 2 — a~^, then the only (a, /5)- winning set is X itself. 
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However, if /3 > 2 — a~^, then there can be proper subsets of X which 
are (a, winning. Schmidt introduced these notions in his study of badly 
approximable numbers and related Diophantine problems. Schmidt games 
have proved to be very useful in showing that certain sets which naturally 
arise in dynamical systems and which are small in the conventional sense are 
indeed large in the sense of Hausdorff dimension. 

We state without proofs some basic properties of a-winning sets: 

Proposition 4. (i) Suppose S, S' are a-winning sets in X, X' then S x S' (Z 
X X X' is also an a-winning set. 

(a) Let Xi, 1 < i < k, be closed suhspaces of a compact metric space X 
whose interiors cover X . If Si C Xi is ai-winning in Xi then S = Ui<i<fc 'S'i 
is a winning where a = minjaj | 1 < i < k}. 

(Hi) (Prop. 3.3[jl|, cf. §11,|T0[) Any a-winning set S in a complete Rie- 
mannian manifold M is large, i.e., for any nonempty open set U <Z M the 
Hausdorff dimension of S (1 U is n where n = dimM. 



(iv) (0, cf. Th. 1, |T^) Let f : X — >Y be a homeomorphism between 



complete metric spaces and X be a bi-Lipschitz constant for f , that is, A > 
1 is a real number such that for any x,y G X, one has X^^dx{x,y) < 
dY{f{x),f{y)) < Xdx{x,y). If S C X is {a, (3) -winning set in X with 
[3X^ < 1, then f{S) is an {a/ X"^, (3 X^) -winning set in Y. In particular, if 
S is a-winning, then f{S) is a/ -winning. □ 

Remark 5. If X,Y are smooth compact Riemannian manifolds and / : 
X — >Y is a diffeomorphism, then a bi-Lipschitz constant for / always exists 
(see Ch. 5 of @], Prop. 3.3 of Thus, in view of Prop. H(if), the 

'largeness' of a subset of a compact Riemannian manifold does not depend 
on the specific choice of the metric. 

Remark 6. Let A be any lattice in the Euclidean space M"' and let T = M"/A. 
Consider the collection Cj. Then there exists a subset 5* C T which is 
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exceptional relative to Cf such that S is a- winning for any a G (0,1/2]. 
In particular, S (1 U has Hausdorff dimension n for any nonempty open set 
f/ C T. To see this, note that if A = Z" C M", then this is just a restatement 
of Dani's result 0. In the general case, we observe that Dani's proof still 
goes through when Z" is replaced by an arbitrary uniform lattice A. We can 
also see this easily using Dani's result for the torus M^/Z" and Remark |^. 

We conclude this section with the following 

Lemma 7. Let p : M — >M' be a smooth fibre bundle where M and M' 
are compact Riemannian manifolds. Assume that S' C M' is a-winning for 
any a G (0, z/] for some v G (0,1/2]. Then there exists a fj, > such that 
S ■.= p^^{S') is a-winning for any a G (0,/x]. 

Proof: Let F be the fibre of the bundle, endowed with a Riemannian metric. 
Let V C M' be any trivializing open set which is diffeomorphic to the unit 
ball B in M". Let K be the image in V of the closed disk of radius 1/2 under 
the diffeomorphism B = V. Let hy '■ P^^{V) — >V x F be a diffeomorphism 
which is a trivialization. Then S'^ := S'dK is a.n a- winning set in K for any 
a G (0, u] and so, by Prop, ^(i), S*^ x F is an a-winning set in K x F for any 
a G (0, z/]. Now, it follows from Prop. |(w) that p~^{S'j^) = hy^{S'j^ x F) is 
a-winning in p'^lK) for any a G (0, z^/A^] where A is a bi-Lipschitz constant 
for the diffeomorphism hy- One can cover M' by the interiors of (closed) disk 
neighbourhoods Ki, 1 < i < k, where Ki G Vi G M', the Vi being trivializing 
open sets for the bundle p : M — >M'. Choose bi-Lipschitz constants Aj for 
trivializations p~^{Vi) — >Vi x F. Then p^^{S'j^.) is a-winning in p^^{Ki) for 
any a G (0, z//A^] and for each i such that 1 < i < k. Hence, by Prop. §(m), 
it follows that S = [jKi^kP^^i^Ki) is a-winning for any a G (0,/i] where /i 
is the smallest of the numbers u/X^, 1 < i < k. □ 
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3 Proofs of Main Theorems 

Let / : X — >X be a self- map and let x & X. Denote by Cf{x) the closure 
of the set {f^{x) | A; > 0}. 

Lemma 8. Let p : X — >X' be any continuous map. Let C and C be any 
collections of self-maps of X and X' respectively such that given f & C, there 
exists an f e C such thatpo f — fop. Suppose that S' C X' is exceptional 
relative to C , then S :— p~^{S') is exceptional relative to C. 

Proof: Let x E S, f E C. Suppose that y e Cf{x) is /-periodic, say, 
/^(y) = y. Let x' = p{x) e 5" and let y' = p{y). Choose /' e C such that 
pof = fop. Then, p{Cf{x)) C C/'(x'). In particular y' e Cf,{x'). Note that 
f^[y') — f^ op[y) — po f^[y) — p[y) — y' which contradicts our hypothesis 
that S' is exceptional. This shows that S is exceptional relative to C. □ 

Lemma 9. Let p : X — >X be a finite covering projection where X, X are 
compact metric spaces. Let C and C be any collections of self-maps of X 
and X respectively such that for any f & C, there exists an f & C such that 
po f — f op. Let S be an exceptional set relative to C. Then S :— p{S) C X 
is an exceptional set relative to C. 

Proof: Let x E S, f E C and let y e C^{x). We shall assume that y is an 
/-periodic point and arrive at a contradiction. 

Since y G C^{x), there exists a sequence of positive integers such 
that converges to y. Choose x E S. f E C such that p{x) = x and 

p o f = f o p. Consider the set {/"'=(a;)|/i; > 1} C Cf{x). Because x E S, this 
set is infinite. Since X is compact, it has a subsequence /"'=»(x) converging 
to an element y E X and y G Cf{x) since Cf{x) is closed. The element y 
must be in the fibre over y E X by continuity of p. Since y is /-periodic, and 



8 



since / covers /, it follows that y must be /-periodic. That is, y G Cf{x) is 
an /-periodic point which is a contradiction since x E S, f E C and 5* is an 
exceptional set relative to C □ 

We are now ready to prove the main theorems stated in the introduction. 
We keep the notations of §1. 

Proof of Theorem ^: Consider the fibration p : M — >M' where M' is the 
torus N/A and let C = Cj. By Remark ^ there exists an exceptional set 
S' C M' relative to C which is a-winning for any a G (0, 1/2]. Note that 
for the collection C of the theorem, the hypotheses of Lemma |^ are satisfied. 
It follows that S = p^^{S') is exceptional. By Lemma |^, we see that S is 
a-winning for any a G (0, /i] for some /i G (0, 1/2]. □ 

Proof of Theorem ^: Consider the finite covering projection vr : M — >M. 
Recall that the deck transformation group G = T/A acts on M by isometrics. 

The hypotheses of Lemma ^ holds when we take C to be as in Theorem 
^ that is, the set of all surjective endomorphisms p of M such that p G Cj. 

By Theorem ^ there exists a positive number p < 1/2 and a set S which 
is exceptional relative to C such that S is a- winning for any a G (0,/i] when 
M is given the right invariant metric. By Prop, ^{iv) applied to the identity 
map of M, the same statement holds if we replace /i by a suitably smaller 
positive number u when we put the G-invariant metric on M. By Lemma |^, 
S := 7r{S) is an exceptional set relative to C. 

Since G acts on M by isometrics, the covering projection vr is a local isom- 
etry. Choose a finite collection {K} of normal (closed) disk neighbourhoods 
whose interiors cover M and such that nlK : K — >tt{K) is an isometry for 
each K. Then, fl is a- winning in K and 7r(S' fl K) is a-winning for each 
a G (0, z/] by Theorem 1, UTOf. It follows from Prop. that S = 7^(3) is 

an a-winning set for any a G (0, z/]. This completes the proof. □ 
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Concluding Remarks: (i) It would be interesting to know whether there 
are large exceptional sets for the class consisting of an arbitrary expanding 
endomorphism (cf. 0) of a nilmanifold or an infranil manifold. 

(ii) Kleinbock |^ has obtained a generalization of Dani's theorem showing 
existence of large sets in a very general setting. However, Kleinbock's result 
still does not subsume our results because the exceptional set in his set up 
is not known to be an a- winning set. This is a stronger conclusion in our 
case which enables us to deduce the maximality of Hausdorff dimension for 
exceptional sets of a family of endomorphisms and not just for exceptional 
sets of a single endomorphism. 
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